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With  our  nation  engaged  in  the  historic 
Global  War  on  Terrorism,  every  ele¬ 
ment  of  our  national  security  apparatus 
must  maintain  a  sharp 
focus  on  how  to  best  aid  this  effort.  Test 
and  evaluation  (TScE)  is  no  different  in 
this  regard  and  is  doing  many  things  with 
the  single  aim  of  getting  effective  and 
suitable  weapon  systems  to  our  warfight¬ 
ers  in  the  field  as  quickly  and  efficiently  as 
possible.  To  accomplish  these  goals,  the 
T&E  community  is  engaged  on  two 
fronts:  first,  we  are  responding  to  today’s 
call  for  urgency  in  all  we  do  to  support  the 
warfighter’s  operational  needs;  and  sec¬ 
ond,  we  are  working  to  effect  lasting  pol¬ 
icy  changes  that  will  result  in  more  reli¬ 
able  future  defense  systems. 

To  ensure  that  systems  designed  to  meet  urgent 
operational  requirements  are  deployed  as  quickly  as 
possible  and  with  as  much  confidence  as  possible,  the 
T&E  community  has  developed  and  is  utilizing  orga¬ 
nizational  commitment — and  individual  creativity — 
and  flexibility.  These  characteristics  are  necessary 
ingredients  because  requirements  and  concepts-of- 
operations  terms  for  urgently  needed  systems  are 
often  vaguely  stated.  Many  times,  the  requirement  is 
simply  boiled  down  to  “something  better  than  what 
we  have.”  Another  characteristic  of  urgent  opera¬ 
tional  requirements  is  shortened  timelines,  which 
often  means  fielding  without  the  usual  breadth  of 
testing,  because  when  threat  operational  modes 
change,  our  engaged  warriors  must  have  responsive 
systems  immediately.  When  all  of  these  factors  are 
considered,  it  is  obvious  to  see  why  the  T&E  com¬ 
munity  is  working  so  hard  to  react  to  and  accommo¬ 
date  shortened  timelines  that  are  consistent  with 
meeting  urgent  requirements. 

Why  is  it  so  important  for  us  to  ensure  that  our 
warfighters  can  employ  new  capabilities  as  quickly  as 
possible  and  with  as  much  confidence  as  possible? 


Because  if  a  system  cannot  be  counted  on  to  perform 
when  needed,  not  only  is  mission  success  jeopardized, 
but  our  warriors  will  develop  doubts  about  the  weapon 
system’s  performance,  which  can  impact 
both  individual  and  organizational  mission 
performance. 

The  T&E  community  has  responded  to 
these  challenges  by  sometimes  working 
around  the  clock  at  ranges  such  as  Yuma 
(Arizona)  and  Aberdeen  (Maryland)  to 
provide  24-  or  48-hour  turnarounds  for 
information  on  critical  equipment  and  sys¬ 
tems,  such  as  body  and  vehicle  armor. 

Testers  are  also  working  to  help  meet 
the  urgent  needs  of  our  warfighters  in  the 
critical  mission  of  defeating  Improvised 
Explosive  Devices  (lEDs).  Eor  this 
urgent  need,  the  Army  Test  and 
Evaluation  Command  (ATEC)  has  taken  on  the  mis¬ 
sion  to  plan,  conduct  and  report  the  results  of  tests, 
simulations,  experiments  and  evaluations  to  ensure 
that  our  warfighters  have  the  right  capabilities  for  suc¬ 
cess  across  the  entire  spectrum  of  operations.  As  part 
of  these  efforts,  testers  at  ATEC  are  conducting  rapid 
testing  in  direct  support  of  the  warfighter  to  provide 
information  on  the  capabilities  and  limitations  of 
untested  weapon  systems  issued  directly  to  our  soldiers 
conducting  combat  operations.  The  Joint  lED  Defeat 
Organization  (JIEDDO)  expects  testers  to  use  flexi¬ 
ble,  streamlined  and  tailored  test  procedures  that  are 
based  on  standard  test  protocols.  Examples  include: 
reusing  knowledge  and  data  from  other  projects;  shar¬ 
ing  data  among  services  and  agencies;  and  providing 
concise  and  timely  reports  to  enable  decisions  on  field¬ 
ing,  improvement  or  termination. 

Like  the  Army,  the  Air  Eorce  T&E  community  is 
working  hard  to  be  responsive  to  the  urgent  operational 
needs  of  our  warfighters,  and  is  providing  rapid  evalua¬ 
tions  of  components  for  urgently  needed  capabilities 
such  as  Integrated  Base  Defense  Security,  and  Global 
Hawk  and  Small  Diameter  Bomb  employment. 


• 

J.  ► 


Dr.  Charles  E.  McQueary 
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The  Navy  T&E  community  also  is  responding 
across  the  entire  spectrum  of  urgently  needed  warfight¬ 
er  capahilities,  including  efforts  to  evaluate  and  provide 
information  on  the  Counter-Bomh/Counter-Bomher 
Advanced  Concept  Technology  Demonstration,  which 
will  help  meet  evolving,  asymmetrical  and  sophisticat¬ 
ed  terrorist  threats.  These  detection  and  mitigation  sys¬ 
tems  will  provide  force  protection  personnel  with  the 
latest  concept  of  operations,  tactics,  techniques  and 
procedures,  as  well  as  rules  of  engagement  generation, 
update  and  dissemination. 

Another  way  that  testers  are  meeting  the  challenges 
of  urgent  operational  requirements  is  through 
deployed  Forward  Operational  Assessment  (FOA) 
teams.  FOA  teams  ensure  that  information  needs  are 
identified  to  state-side  ranges  and  that  the  informa¬ 
tion  supplied  from  the  ranges  is  relevant  and  properly 
reported  to  our  forces.  This  response  hy  the  TScE 
community  is  especially  noteworthy  because  it  often 
involves  a  level  of  commitment  and  sacrifice  that  is 
outside  the  typical  TScE  operating  envelope,  and  it 
exemplifies  the  ends  to  which  the  T&E  community  is 
prepared  to  go  to  support  our  warfighters. 

The  second  front  on  which  the  T&E  community  is 
engaged  is  the  policy  arena:  to  effect  lasting  policy 
changes  that  will  result  in  more  reliable  future  defense 
systems.  This  front  involves  efforts  in  the  Pentagon,  by 
the  offices  of  the  Director,  Operational  Test  & 
Evaluation  (DOT&E)  and  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics 
(USD[AT&L]),  to  develop  and  implement  policies 
that  will  improve  the  reliability  of  systems  that  reach 
our  warfighters  through  the  traditional  or  formal  acqui¬ 
sition  process. 

This  effort  is  important  because  a  large  number  of 
programs  are  not  satisfactorily  demonstrating  that  they 
meet  reliability  requirements,  and  poor  reliability  drives 
down  mission  accomplishment  and  affordability,  as  well 
as  drives  up  force  structure,  total  cost  of  ownership  and 
the  logistics  footprint.  This  means  that  if  a  weapon  sys¬ 
tem  is  dependent  upon  a  component  that  is  prone  to 
failure,  the  affected  military  service  must  plan,  organize, 
equip  and  operate  in  a  manner  to  offset  or  accommo¬ 
date  the  expected  component  failure.  The  resulting 
logistical  footprint  of  components  needed  to  replace  a 
failed  system  include:  the  creation  of  redundant  systems 
and  the  provision  of  spares  that  must  be  purchased, 
transported,  stored  and  used  to  repair  the  system.  These 
steps  may  require  added  personnel,  tools,  test  equip¬ 
ment  and  training  (or  contract  support)  to  find  and 
replace  the  failed  component  and  to  return  the  system 


to  service.  Obviously,  it  is  preferable  to  minimize  oper¬ 
ational  down-time,  costs  and  logistical  footprint  that 
result  from  insufficient  system  reliability. 

Another  frustrating  issue  that  testers  contend  with 
in  the  reliability  area  is  systems  coming  to  test  before 
they  are  sufficiently  mature.  Often,  schedule  consid¬ 
erations  preempt  methodical  system  technical  devel¬ 
opment,  and  tests  can  begin  while  the  system  is  stiU 
technically  immature.  Sufficient  maturity  in  reliabili¬ 
ty  is  especially  troubling,  as  test  experience  indicates. 
Looking  at  initial  operational  T&E  (lOT&E)  results, 
for  which  DOT&E  reports  to  Congress,  the  trend  is 
not  good.  From  2001  to  2005,  the  suitability  per¬ 
formance  rate  for  systems  ranged  from  33  percent  to 
60  percent  of  systems  found  not  suitable  in  lOT&E. 
In  the  Army,  from  1996  to  2006,  66  percent  of  sys¬ 
tems  failed  to  meet  reliability  requirements  and,  of 
those,  75  percent  failed  to  meet  one-half  of  the 
requirement.  Navy  testing  in  Fiscal  Year  2007  result¬ 
ed  in  more  than  half  of  its  systems  being  found  not 
suitable.  A  Fiscal  Year  2007  breakdown  of  DT/OT 
events  indicates  that  43  percent  of  Air  Force  tests 
were  slipped.  In  Fiscal  Year  2006,  25  percent  (two  of 
eight)  of  the  OTs  that  were  started  had  to  be  stopped 
because  of  system  deficiencies. 

These  unsatisfactory  results  add  up  to  our  forces  not 
receiving  the  planned  equipment  improvements  they 
need,  because  either  the  system  fielding  had  to  be 
delayed  to  fix  suitability  problems — most  often  the 
deficiencies  are  in  reliability — or  the  choice  was  to  field 
the  system  with  its  built-in  lack  of  reliability.  We  know 
reliability  deficiencies  can  compromise  mission  effec¬ 
tiveness,  because  as  I  mentioned  previously,  if  an  unre¬ 
liable  system  cannot  be  counted  on  to  perform  when 
needed,  then  our  warfighters  will  obviously  doubt  the 
system’s  performance,  and  this  will  erode  overall  mis¬ 
sion  success. 

Deficiencies  mean  that  part(s)  of  the  system  need  to 
be  reworked  to  get  what  was  originally  desired.  In  my 
experience  in  the  private  sector,  it  is  this  “rework”  that  is 
the  big  cost  and  schedule  driver. 

So,  what  can  the  test  community  do  to  reduce  the 
rework  problem?  First  and  foremost,  the  deficiencies 
must  be  discovered  and  corrected  as  early  as  possible — 
lOT&E  is  too  late.  I  have  said  that  “OT^E  should  be  a 
time  of  confirmation,  not  discovery.  ’’This  is  only  possible 
if  the  systems  engineering  work  of  which  reliability 
design  and  testing  is  an  integral  part  is  done  up  front. 
Emphasis  on  reliability  testing  needs  to  go  beyond  tra¬ 
ditional  tasks  (that  is,  failure  modes  and  effects  analysis, 
failure  reporting  analysis  and  corrective  action  system. 
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environmental  stress  screening  and  burn-in,  and  so 
forth)  to  include  physics  of  failure  modeling  and  sim¬ 
ulation  (that  is,  fatigue  analysis  tools,  finite  element 
modeling,  dynamic  simulation,  heat  transfer  analyses 
and  so  forth)  and  reliability  design  testing.  Reliability 
design  testing  includes  Reliability  Enhancement 
Testing/Highly  Accelerated  Life  Testing  (HALT), 
Accelerated  Life  Testing,  and  environmental  survey 
tests.  Without  comprehensive  lower-level  testing  (for 
example,  HALT)  on  most  or  all  critical  subassemblies, 
and  without  significant  integration  and  developmental 
testing,  it  is  unlikely  that  high  levels  of  reliability  will 
be  achieved. 

The  significance  of  the  suitability  problem  can  be 
defined  and  measured  in  terms  of  the  cost  over  the  life 
cycle  and  the  cost  to  correct  the  problem  if  it  moves  past 
the  design  process.  This  is  why  DOTScE  has  made 
improving  suitability  its  first  priority  and  has  sponsored 
studies  to  determine  the  costs  of  unreliability,  as  well  as 
the  return  on  investment  (ROI)  that  is  possible  with  tar¬ 
geted  reliability  engineering.  The  results  are  striking. 

One  study  found  that  reliability  goals  did  not 
appear  to  be  informed  management  decisions  or  the 
initial  engineering  effort.  Another  study  also  found 
significant  re-design  effort  after  the  initial  design  was 
complete  and  the  systems  were  through  their 
lOT&Es.  Lor  the  purpose  of  the  studies,  the  late 
effort  to  correct  problems  had  a  special  benefit,  name¬ 
ly,  the  budget  justifications  called  out  the  amounts 
spent  correcting  reliability  problems  after  develop¬ 
mental  and  operational  tests.  With  the  ability  to  track 
the  budget  to  improve  reliability  and  the  ability  to  cal¬ 
culate  the  reduced  life  cycle  operating  and  support  costs, 
each  group  could  calculate  the  ROI.  The  ROIs  for 
investment  in  reliability  improvements  were  all  greater 
than  4:1.  The  median  reliability  ROI  was  15:1. 

In  2005,  a  representative  of  the  Defense  Supply 
Center  reported  that  the  Defense  Logistics  Agency 
(DLA)  Reliability  program  had  similar  results.  The 
programs  were  smaller  and  focused  on  component 
improvements  for  aviation  components  in  already  field¬ 
ed  systems.  In  total,  DLA  provided  S36  million  from 
Liscal  Year  2003  to  Liscal  Year  2005,  for  which  it  esti¬ 
mates  a  10-year  savings  of  S496  million,  broken  down 
as  follows: 

■  Twenty-seven  Army  projects  with  S14.1  million 
funding  with  1187  million  in  estimated  savings; 

■  Lorty-five  Navy  projects  with  S9.7  million  funding 
and  S207  million  in  estimated  savings;  and 

■  Twenty-one  Air  Lorce  projects  with  $8.3  million 
funding  and  1102  million  in  estimated  savings. 


While  these  projects  were  small,  a  13.8  ROI  is  con¬ 
sistent  with  the  findings  for  the  larger  and  system-level 
findings  of  our  studies. 

DoD’s  Reduction  of  Total  Ownership  Costs  (R- 
TOC)  program  is  a  slightly  different  example  of  what 
ROIs  are  possible.  Liscal  Year  2005  cost  avoidances 
were  reported  to  be  $2.1  billion,  and  projected  life 
cycle  cost  avoidances  will  exceed  $76  billion.  The  R- 
TOC  program  is  Office  of  the  Secretary  of  Defense 
(OSD)-led  and  Service-supported.  It  seeks  to  facili¬ 
tate  both  new  technologies  and  management  practices 
to  improve  readiness  and  reduce  ownership  cost. 
Lunding  for  R-TOC  was  provided  from  DoD 
through  out-of-POM  cycle  sweep-up  funding 
through  three  Program  Budget  Decisions  (PBDs). 
The  Services  were  not  required  to  set  up  and  fund  R- 
TOC  projects.  The  DoD  funding  total  was  approxi¬ 
mately  $40  million  in  these  PBDs.  So  the  billions  in 
savings  resulted  from  an  investment  of  approximately 
$40  million,  plus  intensive  management  attention  to 
improve  techniques  and  procedures. 

The  R-TOC  program  addresses  already  existing  sys¬ 
tems.  What  ROIs  are  possible  if  the  investment  is  done 
as  part  of  the  up-front  systems  engineering  effort?  In 
the  cases  where  the  investment  occurred  after  the  initial 
design  was  complete  and  often  after  the  lOT&E,  cal¬ 
culating  the  ROI  was  possible  because  the  reliability 
improvement  occurred  after  the  initial  design  was  com¬ 
plete,  so  the  cost  was  separate  from  the  initial  systems 
engineering  effort.  StiU,  the  reduction  in  operations  and 
sustainment  (O&S)  cost  was  in  the  23-  to  86-percent 
range.  There  is  evidence  that  an  80-percent  reduction  in 
O&S  is  not  unreasonable  to  expect  when  the  life  cycle 
support  costs  are  considered  from  the  very  beginning 
and  design  choices  made  accordingly.  In  a  simulation  of 
a  complex  ground  vehicle  electronics  system,  one  study 
found  that  reliability  improvement  is  the  most  signifi¬ 
cant  factor  in  reducing  the  cost  of  ownership  among  the 
factors  examined.  The  factors  were  reliability,  common¬ 
ality  of  parts  (number  of  different  configurations 
allowed),  support  system  performance,  packaging  (that 
is,  the  use  of  line-replaceable  modules  versus  line- 
replaceable  units)  and  maintainability.  However,  when 
all  factors  were  considered  together  and  assuming  best 
possible  trades  were  made,  an  80-percent  reduction  in 
overall  support  cost  might  be  possible — ^which  can  pro¬ 
duce  biUion-doUar  savings. 

In  one  case  (Lorce  XXI  Battle  Command,  Brigade- 
and-Below),  the  improvement  in  reliability  decreased 
the  expected  20-year  support  cost  from  $13.6  billion  to 
$1.8  billion  (an  86-percent  reduction).  In  another  case. 
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the  Air  Force  post-IOT&E  investment  in  reliability 
improvements  for  the  F-22  are  expected  to  reduce  the 
costs  from  S30  billion  to  S25  billion,  based  on  the  cur¬ 
rent  expected  buy  of  182  total  active  inventory  aircraft 
(with  a  148  primary  aerospace  vehicle  authorization) 
and  a  24-year  life.  If  the  buy  were  bigger  or  the  life 
extended  (as  is  often  the  case  for  defense  systems),  the 
savings  would  be  greater. 

Because  efforts  and  investments  to  improve  reliabil¬ 
ity  must  happen  before  IOT8cE  and  in  the  early  devel¬ 
opmental  phases,  DOTScE  and  USD(AT8cL)  have 
developed  a  key  partnership  that  is  working  to  identify 
and  exploit  opportunities  to  formulate  new  policies  for 
improving  reliability.  These  joint  efforts  have  so  far 
included:  sponsorship  of  an  updated  DoD  Reliability, 
Availability  and  Maintainability  (RAM)  Handbook: 
joint  sponsorship  of  a  congressionaUy  mandated  report 
on  current  TScE  policies  and  practices  and  how  they 
can  be  improved:  and  joint  sponsorship  of  a  Defense 
Science  Board  taskforce  on  Developmental  TScE 
(DT&E)  that  win  examine  how  to  strengthen  DTScE 
oversight.  DOTScE  and  USD(AT8cL)  are  also  working 
together  to  provide  some  guidance  for  the  development 
of  the  mandatory  Key  Performance  Parameter  for 
Materiel  Availability. 

Working  with  the  Joint  Staff  and  USD(ATScL),  we 
have  initial  agreement  on  a  method  for  developing  and 
justifying  RAM  requirements.  This  will  provide  a 
sound  basis  for  the  requirement,  and  the  logistical  con¬ 
ditions  for  system  employment  that  would  be  a  valid 
basis  for  test.  Another  ongoing  joint  effort  is  to  look  at 
the  system  development  contracting  process — specifi¬ 
cally  the  request  for  proposal  to  industry,  and  determine 
how  to  include  reliability  requirements  in  the  system 
development  statement  of  work. 

So  these  are  just  a  few  of  the  ways  that  the  TScE 
community  has  answered  the  call  to  remain  focused  on 
our  brave  airmen,  soldiers,  sailors  and  marines  as  they 
sacrifice  and  risk  all  to  defend  our  nation  against  ene¬ 
mies  that  are  determined  to  undermine  and  destroy  it. 
To  me,  the  striking  characteristic  of  the  national  secu¬ 
rity  community  of  which  TScE  is  a  part  is  how  the  old 
parochialisms  seem  to  melt  away  as  we  all  become  unit¬ 
ed  and  focused  on  supporting  our  warfighters.  It  is  in 
these  times  that  we  cease  to  become  developmental 
testers,  or  operational  testers,  and  instead  become  just 
plain  testers,  looking  for  the  best  way  to  meet  the 
urgent  operational  needs  of  our  men  and  women  at  war, 
and  to  do  what  makes  the  most  sense  to  deliver  them 
the  best  systems  possible.  In  the  final  analysis,  we  do 
not  serve  the  acquisition  community;  we  serve  the  men 


and  women  who  are  protecting  our  freedoms  with  the 
equipment  we  deliver  to  them.  □ 
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ing  policies  and  procedures  for  the  conduct  of  operational 
and  live  fire  test  and  evaluation.  Prior  to  this  appointment. 
Dr.  McQueary  was  the  first  Under  Secretary  for  Science 
and  Technology  at  the  Department  of  Homeland  Security 
and  confirmed  by  the  US.  Senate  in  March  of 2003.  In  this 
position,  he  led  the  research  and  development  arm  of  the 
department,  utilizing  the  nations  scientific  and  technologi¬ 
cal  resources  to  provide  federal,  state  and  local  officials  with 
the  technology  and  capabilities  to  protect  the  homeland.  Dr. 
McQueary  is  a  former  president  of  General  Dynamics 
Advanced  Technology  Systems,  Greensboro,  North 
Carolina.  He  also  has  been  president  and  vice  president  of 
business  units  for  AT^T,  Lucent  Technologies,  and  director 
for  ATIAT  Bell  Laboratories.  Early  in  his  career  at  Bell 
Laboratories,  Dr.  McQueary  served  as  head  of  the  Missile 
Operations  Department  for  the  SAFEGUARD 
Antiballistic  Missile  Test  Program,  based  at  Kwajalein 
Atoll  in  the  Marshall  Islands.  He  later  headed  Bell 
Laboratories’  Field  Operations  Department  in  Great 
Britain  in  support  of  a  Navy  oceanographic  research  station. 
He  also  served  as  the  director  of  Undersea  Systems 
Development  Lab.  Dr.  McQueary  is  a  former  executive 
board  member  of  the  National  Security  Industrial 
Association  and  the  American  Defense  Preparedness 
Association  (ADPA)  (combined  to  form  the  National 
Defense  Industrial  Association  [NDIA]).  He  is  a  past 
chairman  of  the  Undersea  Warfare  Systems  Division  of 
ADPA  and  a  former  member  of  the  Navy  League  Industrial 
Executive  Board,  the  Navy  Submarine  League,  the 
Electronics  Industries  Association,  the  American  Society  of 
Mechanical  Engineers  and  the  American  Association  for  the 
Advancement  of  Science.  He  is  also  the  recipient  of  the 
NDIA  Homeland  Security  Leadership  Award.  Dr. 
McQueary  is  a  graduate  of  The  University  of  Texas, 
Austin,  where  he  earned  a  bachelor  of  science  degree  in 
mechanical  engineering;  master  of  science  degree  in  mechan¬ 
ical  engineering;  and  a  Ph.D.  in  engineering  mechanics  as  a 
National  Aeronautics  and  Space  Administration  (NASA) 
Scholar  (masters  of  science  and  Ph.D.)  and  member  of  five 
academic  honor  societies.  The  University  of  Texas  has 
named  Dr.  McQueary  a  Distinguished  Engineering 
Graduate. 
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